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ADVANCED CONTROL DEVICE FOR HOME ENTERTAINMENT UTILIZING 

THREE DIMENSIONAL MOTION TECHNOLOGY 

The present invention relates to the control of 
home entertainment devices and applications/ and more 
particularly, to a method and system for controlling and 
transferring data to home entertainment devices by 
manipulating a control device. 

Hand-held devices, such as remote controls 
devices, are typically used to control consumer electronic 
devices, such as televisions and gaming machines. As the 
hand-held devices and consumer electronic devices have 
become more sophisticated, new techniques for inputting 
commands to the hand-held devices have been developed. 
These techniques include methods that detect the 
orientation of a hand-held device to generate a command. 
For example, U.S. Patent Numbers 4,745,402 and 4,796,019 
disclose methods for controlling the position of a cursor 
on a television. U.S. Patent Number 6,603,420 discloses a 
remote control device that detects the direction of 
movement of the remote control device to control/ e.g., the 
channel and volume selection of a television. 

The ability of these hand-held devices to hold 
data and the development of more sophisticated capabilities 
in the consumer electronic devices has created new 
challenges for controlling these consumer electronic 
devices. For example, it is often necessary to transfer 
data from the hand-held device to the consumer electronic 
device or vice versa. The hand-held device should also 
provide a natural, efficient mechanism for indicating that 
an action, such as a data transfer, is to be performed. A 
need therefore exists for an improved hand-held device that 
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is capable of efficiently generating commands and 
transferring data to or from consumer electronic devices. 

An apparatus and method are disclosed for 
generating commands and transferring data between a hand- 
held device and a base device (including consumer 
electronic equipment) . The hand-held device is capable of 
detecting the motion of the hand-held device itself, 
interpreting the motion as a command, and executing or 
transferring the command. The motion of the device can 
include gestures made by the user while holding the device, 
such as the motion of throwing the hand-held device toward 
a base device, as a user would do when swinging a tennis 
racket. The commands generated by the user range from 
basic on/off commands to complex processes, such as the 

transfer of data. 

In one embodiment, the user can train the device 
to learn new motions associated with existing or new 
commands. For example, the user can make the motion of 
throwing the hand-held device toward the base device. The 
hand-held device analyzes the basic components of the 
motion to create a motion model such that the motion can be 
uniquely identified in the future. 

A more complete understanding of the present 
invention, as well as further features and advantages of 
the present invention, will be obtained by reference to the 
following detailed description and drawings . 

FIG. 1 shows an exemplary hand-held device of the 

present invention; 

FIGS. 2A-B illustrate gestures that are 
interpreted as commands by the hand-held device of FIG. 1; 
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FIG. 3 is a schematic block diagram of the hand- 
held device of FIG, 1; 

FIG. 4 illustrates an exemplary embodiment of a 
motion detection subsystem; 

FIG. 5 is a flowchart describing an exemplary 
implementation of the system process of the hand-held 

device of FIG. 1; 

FIG. 6 is a flowchart describing an exemplary 
implementation of a motion training process; FIG. 7 is a 
flowchart describing an exemplary implementation of a 
motion detection process; and 

FIG. 8 is a graph illustrating the motion model 
of a throwing motion based on the expected acceleration in 
each of three perpendicular planes. 

FIG. 1 shows an exemplary hand-held device 300 of 
the present invention, discussed further below in 
conjunction with FIG. 3, such as the Philips Super Pronto, 
modified in accordance with the features of the present 
invention. The hand-held device 300 is capable of 

detecting motion of the hand-held device 300, interpreting 
the detected motion as one or more commands, and executing 
or transferring the command (s). 

FIGS. 2A-B illustrate gestures that a user can 
make using the hand-held device 300. FIG. 2A, for example, 
shows a user 201 making the gesture of throwing the device 
300 toward a base device, such as television 210. FIG- 2B 
shows a user making the gesture of pouring from the device 
300 into a base device, such as television 210. The 
gesture and associated motion indicate that the user 201 
would like to transfer data from the hand-held device 300 
to the television 210. In this case, the user would first 
locate and identify the data (e.g. a picture or music) and 
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then make the gesture toward the base device. The data 
could be identified, for instance, by selecting an item 
from of a list displayed on the hand-held device 300. The 
data would then be transferred. In addition, if the data 
is a picture, it could be (optionally) displayed on the 
television or, if the data is music, it could be 
(optionally) played through the speakers. Other gestures 
include making a pulling motion (not shown) directed from a 
base device towards the user. In this case, the gesture 
would indicate that the identified data should be 
transferred to the hand-held device 300. The data would 
then be retrieved from either the base device itself, or 
from another device (e.g. a server) . Since there are a 
number of base devices 210 through 214 located in the area 
of the user 201, the hand-held device 300 has the ability 
to identify which device 210-214 should receive the data 
being transferred (as described in more detail below). FIG. 
3 is a schematic block diagram of an exemplary hand-held 
device 300 of the present invention. As is known in the 
art, the methods and apparatus discussed herein may be 
distributed as an article of manufacture that itself 
comprises a computer-readable medium having computer- 
readable code means embodied thereon. The computer- 
readable program code means is operable, in conjunction 
with a computer system such as central processing unit 301, 
to carry out all or some of the steps to perform the 
methods or create the apparatuses discussed herein. The 
computer-readable medium may be a recordable medium (e.g., 
floppy disks, hard drives, compact disks, or memory cards) 
or may be a transmission medium (e.g., a network comprising 
fiber-optics, the world-wide web, cables, or a wireless 
channel using time-division multiple access, code-division 
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multiple access, or other radio-frequency channel) . Any 
medium known or developed that can store information 
suitable for use with a computer system may be used. The 
computer-readable code means is any mechanism for allowing 
a computer to read instructions and data, such as magnetic 
variations on a magnetic medium or height variations on the 
surface of a compact disk. 

Memory 302 will configure the processor 301 to 
implement the methods, steps, and functions disclosed 
herein. The memory 302 could be distributed or local and 
the processor 301 could be distributed or singular. The 
memory 302 could be implemented as an electrical, magnetic 
or optical memory, or any combination of these or other 
types of storage devices. The term "'memory" should be 
construed broadly enough to encompass any information able 
to be read from or written to an address in the addressable 
space accessed by processor 301. 

As shown in FIG. 3, the memory 302 includes 
motion model database 303, system process 500, discussed 
further below in conjunction with FIG. 5, motion training 
process 600, discussed further below in conjunction with 
FIG. 6, and motion detection process 700, discussed further 
below in conjunction with FIG. 7. Hand-held device 300 
also includes motion detection subsystem 400, discussed 
further below in conjunction with FIG. 4, radio frequency 
(RF) communication subsystem 305, and infrared detection 

subsystem (IDS) 310. 

The RF communication subsystem 305 provides 
communication between the handheld device 300 and one or 
more base devices 210-214 in a known manner. For example, 
the RF communication subsystem 305 may utilize the IEEE 
802.11 standard for wireless communications or any 
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extensions thereof. The IDS 310 emits infrared light in a 
directional manner in order to signal a base device 210-214 
that it should execute the command being transmitted by the 
device 300. Only the base device 210-214 that detects the 
infrared signal should execute the transmitted command. 
The command is transferred to the base device 210-214 via 
the RF communication subsystem 305 in a known manner. In 
an alternative embodiment, the command may be transferred 
by modulating the infrared signal (utilizing, for example, 
the IR Blaster standard) in a known manner. 

FIG. 4 illustrates an exemplary embodiment of 
motion detection subsystem 400. Motion detection subsystem 
400 contains x-axis accelerometer sensor 410, y-axis 
accelerometer sensor 411, z-axis accelerometer sensor 412, 
and corresponding analog to digital converters 415, 416, 
417. Accelerometer sensors 410, 411, 412 detect the 
acceleration of the device 300 along the x-axis, y-axis, 
and z-axis, respectively. The accelerometer sensors 410, 
411, 412 may be embodied, for example, using the 3D Motion 
Sensors commercially available from NECTokin of Union City, 
California. Analog to digital converters 415, 416, 417 
convert the acceleration (s) detected by accelerometer 
sensors 410, 411, 412, respectively, to a digital form that 
can be read by processor 301. In alternative embodiments, 
other components, including stress-sensitive resistive 
elements, tilt sensors, and magnetic direction sensors, may 
be utilized to determine the position, orientation and/or 
speed of movement of the device 300. 

FIG. 5 illustrates an exemplary embodiment of 
system process 500. System process 500 initially waits for 
a command to be entered during step 505. If, during step 
505, a user enters a training command, the system process 
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500 executes step 510 where motion training process 600 is 
called. If, during step 505, a user makes a gesture or 
motion indicative of a command, the system process 500 
executes step 515 where motion detection process 700 is 
called. Upon completion of the called processes 600, 700, 
system process 500 returns to step 505 to wait for the 
entry of a new command. 

FIG. 6 illustrates an exemplary embodiment of 
motion training process 600. Motion training process 600 
learns new gestures and motions demonstrated by a user to 
be used for identifying existing or new commands. For 
instance, a user 201 may want to train the device 300 to 
detect a throwing motion, such as the motion of throwing 
the device 300 toward a television 210. The user first 
presses a switch on the hand-held device 300 to indicate 
that a new gesture is to be created. (Alternatively, the 
user can train the hand-held device 300 to interpret a 
motion as an indication that the training process should be 
executed.) Motion training process 600 initially waits for 
motion to be detected by one or more of the accelerometer 
sensors 410, 411, 412 (step 601) and then records the 
motion detected by the sensors 410, 411, 412 by 
periodically sampling and storing data read from analog to 
digital converters 415, 416, 417 (step 605) , After each 
set of samples have been read during sampling step 605, a 
test is made to determine if no motion has been detected 
for a specified period of time indicating that the gesture 
or motion has been completed (step 608) . If motion is 
detected during step 608, then step 605 is repeated to read 
the next set of samples; otherwise, motion training process 
600 creates and stores a model of the motion captured 
during step 610. The motion model is created in a known 
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manner For example, the following publications describe 
methods for analyzing, comparing and modeling motions and 
gestures: Ho-Sub Yoon, Jung Soh, Younglae J. Bae and Hyun 
Seung Yang, Hand Gesture Recognition Using Combined 
Features of Location, Angle and Velocity, Pattern 
Recognition, Volume 34, Issue 1, 2001, Pages 1491-1501; 
Cristopher Lee and Yangsheng Xu, Online, Interactive 
Learning of Gestures for Human/Robot Interfaces, The 
Robotics Institute, Carnegie Mellon University, Pittsburgh, 
IEEE International Conference on Robotics and Automation, 
Minneapolis, 1996; Mu-Chun Su, Yi-Yuan Chen, Kuo-Hua Wang, 
Chee-Yuen Tew and Hai Huang, 3D Arm Movement Recognition 
Using Syntactic Pattern Recognition, Artificial 
Intelligence in Engineering, Volume 14, Issue 2, April 
2000, Pages 113-118; Ari Y. Benbasat and Joseph A. 
Paradiso, An Inertial Measurement Framework for Gesture 
Recognition and Applications, MIT Media Laboratory, 
Cambridge, 2001; and Mu-Chun Su, Yi-Yuan Chen, Kuo-Hua 
Wang, Chee-Yuen Tew and Hai Huang, 3D Arm Movement 
Recognition Using Syntactic Pattern Recognition, Artificial 
Intelligence in Engineering, Volume 14, Issue 2, April 
2000, Pages 113-118, each incorporated by reference herein. 

The created model will be used to interpret 
future gestures and motions made by the user 201. During 
step 615, the model created during step 610 is assigned a 
command or process that is to be executed when the motion 
associated with the model is detected. The command to be 
executed is identified utilizing well known methods, for 
instance, pressing a switch on the hand-held device 300 
associated with the command or entering a code associated 
with the command on a keypad. In an alternative 

embodiment, the user could enter (record) a series of 
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commands by performing the actions on the system (e.g., on 
the touch screen), similar to recording a macro in MS Word. 
The series of commands can then be associated to a single 
gesture. The assigned command or process is stored with 
the associated motion model in the motion model database 
303. 

FIG. 7 illustrates an exemplary embodiment of 
motion detection process 700. Motion detection process 700 
interprets gestures and motions made by a user 201 to 
determine the command (s) that are to be executed. For 
instance, if the user 201 makes the motion of throwing the 
hand-held device 300 towards the television 210, the hand- 
held device 300 will interpret the gesture as a command to 
transfer data from the device 300 to the television 210. 
Motion detection process 700 initially records the motion 
detected by the accelerometer sensors 410, 411, 412 by 
periodically sampling and storing the data read from analog 
to digital converters 415, 416, 417 (step 705) . After each 
set of samples have been read during sampling step 705, a 
test is made to determine if no motion has been detected 
for a specified period of time indicating that the gesture 
or motion has been completed (step 708) . If motion is 
detected during step 708, then step 705 is repeated to read 
the next set of samples; otherwise, motion detection 
process 700 compares the data collected during step 705 to 
the motion models stored in the device 300 (step 710) , 
During step 710, a score is generated for each model 
comparison. The command or process associated with the 
model that attained the highest score during step 710 is 
then executed during step 715. For example, if the model 
with the highest score was the "'throwing motion" model, 
then a data transfer process (not shown) would be executed 
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in a known manner. The data transfer process can be 
accomplished, for example, utilizing the 802.11 standard in 
a well known manner. During step 720, the IDS 310 is also 
activated, thereby causing an infrared signal to be emitted 
in the direction of the throwing motion. Only the base 
device 210-214 that detects the infrared signal will 
receive the data transferred via the RF communication 

subsystem 305. 

FIG. 8 shows an exemplary motion model 
representing the throwing motion of FIG. 2A. As is 
illustrated, the z-axis accelerometer indicates that the 
motion is in the x-y plane (no motion along the z-axis) . 
As indicated by the x-axis accelerometer, the motion shows 
a quick acceleration along the x-axis, a peak speed at the 
halfway point of the motion, and an increasing deceleration 
as the motion is completed. A similar, but smaller, action 
is occurring along the y-axis. 

It is to be understood that the embodiments and 
variations shown and described herein are merely 
illustrative of the principles of this invention and that 
various modifications may be implemented by those skilled 
in the art without departing from the scope and spirit of 
the invention. 



